Video background subtractor system

ABSTRACT

A video processing system includes at least one video source, a region selecting unit, a subtracting unit and a display unit. The region-selecting unit selects the user-defined region of interest from the video source. The subtracting unit subtracts the required region, selected by the region selecting unit. The output of the subtracting unit is provided to the display unit, which displays the required output. In one embodiment, when video data is received from a plurality of video sources, the selecting of user defined regions of interest from the video sources is supported. The region subtracting unit can be used to subtract the required region of interest from video data and it is displayed on the display unit. In other embodiments of invention, the display unit displays on overlay of two unrelated video streams.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority of U.S. Provisional Patent ApplicationNo. 60/731,254, filed on Oct. 31, 2005. The subject matter of thisearlier filed application is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to real time processing of a video stream.

2. Description of the Related Art

Systems are available in the market which enable users to play backvideo. Systems are available for broadcasting video programs. However,it is not possible for an end viewer to remove or modify some sectionsof the video stream being displayed. The user has to view what isdelivered in its entirety.

Video sources such as a Set-Top-Box (STB), a Digital Video Disk player(DVD), a Personal Video Recorder (PVR), a computer, or cable/satellitebroadcast television, among other sources are often viewed on atelevision (TV) monitor. The video sources provide a wide variety ofprogramming, both live and pre-recorded, that may be presented on a TVor monitor that a viewer selects. Each of these video sources deliversvideo to a TV, and is controlled through direct interaction or via aRemote. To interact with a user, video systems deliver information via aTV's screen and speakers and receive information directly from a Remote.Again, such interaction usually requires a user to look back and forthat the Remote and TV Screen in attempts to locate buttons and understandthe Remote's operation

Different types of video systems are used by a variety of people inenvironments such as homes, offices, schools (educational institutions)or in public places. People typically view their own programs at theirown convenience on these video systems. Similarly, typical home videoenvironment consists of many different members of the family wishing toview their own programs at their own convenience.

Other limitations and deficiencies of the related art can be appreciatedby someone with skill in the art through comparison of the related artto the present invention as described in the reminder of thisspecification and drawing associated herewith.

BRIEF DESCRIPTION OF THE DRAWINGS

For the present invention to be easily understood and readily practiced,preferred embodiments will now be described, for purposes ofillustration and not limitation, in conjunction with the followingfigures:

FIG. 1 is a functional block diagram illustrating the operation of avideo background subtractor system built in accordance with the presentinvention;

FIG. 2 is a flow chart showing exemplary operation of a videotransmission and receiver system wherein the video background subtractorsystem is employed at both the transmitting and the receiving ends;

FIG. 3 is a schematic block diagram illustrating one embodiment of avideo background subtractor system in accordance with the presentinvention;

FIG. 4 is a schematic block diagram that illustrates an exemplaryembodiment of a region selecting unit that acts as a control unit of anexemplary video background subtractor system;

FIG. 5 is a functional block diagram of a video processing systemwherein the video subtracting system used to subtract a region ofinterest from a video data, such as a cable/satellite source, isincorporated into a set-top-box (STB) or into a television, such as theHDTV;

FIG. 6 is a functional block diagram illustrating use of videoprocessing system for overlaying two pure unrelated video streams,wherein a set-top-box (STB) or a television (TV) display comprises avideo background subtractor system and wherein a user can specify aregion of interest using a remote control or keyboard;

FIG. 7 is a schematic block diagram showing an exemplary videobackground subtractor system that is used for transmitting a compositevideo signal incorporating a plurality of video sources; and

FIG. 8 is a schematic block diagram that illustrates an exemplaryembodiment of a set-top-box that comprises the typical components of thevideo background subtraction system, such as a region selecting unit, aregion subtraction unit, a mixing unit.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

FIG. 1 is a functional block diagram illustrating the operation of avideo background subtractor system built in accordance with the presentinvention. The video background subtractor system 105 subtracts userdefined regions of interest from a specified video stream. The videobackground subtractor comprises a first video source 107, a regionselecting unit 109 communicatively coupled to the first video source107, a region subtracting unit 111, for subtracting user defined regionof interest from the first video source 111. The video backgroundsubtractor also comprises a display unit 113 for displaying thesubtracted video data is.

The first video source 107 provides a video data for processing by thevideo background subtractor system 105. The video data that is providedis one of a live TV broadcast from a set-top-box, real-time computergenerated video data from a video game, live video from a image capturedevice, live video from a camcorder, etc. The region selecting unit 109facilitates selection of a region of interest. For example, it providesidentification of a region of interest using an input device such as atouch sensitive screen, a mouse (such as a USB mouse or a wirelessmouse), a wireless pointing device, a touch sensitive pad, etc. Theregion selecting unit 109 makes it possible to identify one or moreregions of interest that are communicated to the region subtracting unit111 for removal from the video data provided by the first video source107.

The region subtracting unit 111 gathers information on one or moreregions to be subtracted from a specified video data. It then subtractsthe regions of interest, adjusts boundary, contrast, shading to make thevideo data compensate for the missing subtracted regions of interest.For example, it replaces the subtracted region of interest with analternative video data that is computed using neighboring regions ofdata that may also be specified by a user or pre-defined as thesuggested replacement. Thus, the region subtracting unit 111 is capableof replacing the regions of interest that are subtracted, if suchreplacements are determined to be necessary.

In one embodiment, the video background subtractor system 105 isconfigured to subtract the background video (i.e. remove the backgroundvideo) while isolating and retaining the identified regions of interest(ROIs) for subsequent processing, such as incorporation into other videodata from another video source.

In one embodiment, the video background subtractor system 105 isincorporated into a video processing system that facilitates processingand enhancements of video data from a plurality of video sources. Thevideo processing system performs video subtraction as well as otherkinds of video data enhancements, such as zooming into some regions ofinterest, or shading the selected region of interest (ROI) differentlyto highlight it as well as increasing it's relative size beforereplacing it in the place of the subtracted region of interest.

In one embodiment, when video data is received from a plurality of videosources, the selecting of user defined regions of interest from thevideo sources is supported. The region subtracting unit can be used tosubtract the required region of interest from video data and it isdisplayed on the display unit. In other embodiments of invention, thedisplay unit displays on overlays ROIs from two unrelated video streamsonto a background video data. In another related embodiment, the displayunit displays a video data created by an overlay of ROIs.

In general, the video background subtractor system 105 separates a ROIfrom a background video when a predefined ROI information is provided orwhen user selection of ROI is employed.

FIG. 2 is a flow chart showing exemplary operation of a videotransmission and receiver system wherein the video background subtractorsystem is employed at both the transmitting and the receiving ends. Theactivities of the video background subtractor system 105 relevant inthis description comprises ROI video identification, video subtractionand video replacement. These activities are distributed upstream anddownstream (remote and local, transmitting and receiving side). Forexample, video identification may occur at a cable TV headend systemwhile the video subtraction occurs downstream in a set-top-box, or evenin the television (TV).

The video background subtractor system 105 operation in the transmittingside is initiated at a block 201 when a core video source startsproviding a video data for processing. The video data from the corevideo source provides information of ROIs that might be identified inthe video data. The core video source is therefore accompanied by ROIsinformation that may be subsequent modified, removed, replaced orenhanced. Then, at a next block 203, the selection of a 1^(st), 2^(nd)and 3^(rd) ROIs from the ROIs provided by the video data, or providedfrom another source (such as a local video source) occurs. At a nextblock 205, the 1^(st) and 2^(nd) ROIs are subtracted. Then, at a nextblock 207, the subtracted 1^(st) ROI is replaced in the video data. Suchreplacements is performed using an enhanced video object created fromthe replaced 1^(st) ROI, using a computed video information (computedusing other ROIs or from video segments in a background determined to berelevant) or using additional ROIs obtained from other sources.

After the replacement, at a next block 209, in the case of a videobroadcast environment or a video origination system, such as a cable TVheadend system, the core video and the ROI information, such as the2^(nd) and 3^(rd) ROI information is communicated to video systems thatare “listening”, i.e. to video systems that are capable of receiving thetransmitted video information. In one embodiment, the video backgroundsubtractor system 105 communicates ROI information to the recipientvideo systems that are capable of processing the ROIs and conductingreplacements and overlays on the video data from the core video sourceemploying the ROIs. In another embodiment, the video backgroundsubtractor system 105 conducts subtraction itself as a source of videodata for a remote recipient video system and communicates video datafrom the core video source modified by the subtraction of the ROIs, suchas the 1^(st) ROI, but it communicates the 2^(nd) and 3^(rd) ROIinformation associated with ROI replacement and overlay.

At the block 209, in the case of a recipient video system that comprisesthe video background subtractor system 105, the received core video maycomprise ROIs and ROI information, such as the 2^(nd) and 3^(rd) ROIinformation. In addition, it receives replacement video for replacingthe 2^(nd) ROI. Being capable of subtraction of ROIs, it subtracts the2^(nd) ROI and replaces it with the replacement video received.

At a next block 211, the recipient video system comprising the videobackground subtractor system 105 selects a 4^(th) ROI. This is supportedusing pre-configured video selection information or by user initiatedselection operation employing input devices. Then, at a next block 213,the 3^(rd) and 4^(th) ROI are subtracted. Then, at a next block 215, the2^(nd), 3^(rd) and 4^(th) ROI are replaced by appropriate replacementvideo (which may be enhanced version of the ROIs themselves). Finally,the processed video that emerges as a combined output is displayed onthe display unit.

FIG. 3 is a schematic block diagram illustrating one embodiment of avideo background subtractor system 105 in accordance with the presentinvention. The system 105 comprises a first video source 307, a regionselecting unit 309 communicatively coupled to the video source, and aregion subtracting unit 311. The video background subtractor system 105also may comprise a mixing unit 319, a second video source 321, and avideo manager 315.

The region selecting unit 309 provides input (user input or predefinedconfiguration information) to the region subtracting unit 311, theoutput of which is connected to the display unit 313. The regionselecting unit 309 is configured to select a particular region ofinterest from the video source 307, for example based upon input from auser. For operation in real time environment, a user can select theregion of interest from the video source while real-time video is beingfed to the selecting unit 309. Utilizing such conventional input andcontrol devices such as keyboard, mouse, wireless pointing device, atablet, a touch screen etc., appropriate parameters are provided, aspart of a selection operation to the video background subtractor system105 that can be used during region selection by the region selectingunit 109.

The appropriate regions of interest are selected based upon appropriatelocating methods such as coordinates in an area of a screen, byestimation of motion vector, block matching technique and by a sum ofsquare difference method and similar region selection techniques. Theselection of a predefined object, whether it is dynamic or static, isbased upon predefined characteristics of the object. Software orhardware can be configured within region selecting unit 309 to track orto follow a dynamic region of interest, such as a talking person, amoving person or moving objects such as a condenser, a racing car orvirtually any other moving device. The subtracting unit 311 can beconfigured to subtract a selected region from the video data providedvia the video source, such as the first video source 307 or the secondvideo source 321.

In one embodiment, a mixing unit 319 is incorporated into the videoprocessing system 305, and is configured to add information fromdifferent video layers, which form the output, after removing one ormore selected regions of interest from the video stream provided by thefirst video source 307. Through the use of an image tracking unitprovided in either the region selecting unit 309 or the mixing unit 119,a moving object can be tracked from the video data from the video source307, 321. Software and hardware are provided that can adjust the bordersof selected region to be removed by the region subtracting unit 311.Further, the mixing unit according to certain embodiments of the presentinvention can apply various transforms to the region of interests beforethe different video layers are added together.

In certain embodiments of the present invention, the display unit 313can be in visual proximity of a viewer viewing the video program,provided as the first video source 307 that is currently displayed onthe display unit 113. The video program can be, for example, abackground scene, a motion picture scene, a video game, etc. The viewerwhose subsequent actions, in response to the video program beingdisplayed is captured as a second video source 321 and communicated tothe mixing unit 319. The video program provided from the first videosource 307 is then processed by the region subtracting unit 311 toremove a ROI associated with a primary character in the video stream(program being displayed or game being played) and the mixing unit 319adds in the viewer's image retrieved as an ROI from the video streamcaptured from the second video source 321 to create an output that isdisplayed on the display unit 313. Such an output is thus been createdby subtracting a region of interest from the first video source 307 andby adding, as a replacement operation, the image of the viewer that isprovided in the video stream provided by the second video source 321.

In one embodiment, the region selecting unit 309 and the mixing unit 319is merged into a single component. In another embodiment, the mixingunit 119 is not employed and the output of the region subtracting unit311, which is the video data received from the first video source 307without the ROIs selected, is displayed on the display unit.

The video data from the video source 307/321 may comprise of a staticbackground with an image of a dynamic object moving on the foreground.Other examples of video data are more motion picture video, video of amartial art display, video games etc. Various video recording can bestored in a video library and accessed by users for various applicationsand the video sources are used to provide such input. The mixing unit319, if needed, is configured to mix various video layers from thedifferent video inputs, based upon specified parameters, which can bepreset by a user.

In one embodiment, an image-tracking unit is incorporated into theregion selecting unit 309, such that tracking of dynamic object in thevideo input from a real time video source becomes possible. It ispossible to provide video data, such as video streams or static videoobjects, from the second video source 121, 321 or even a third videosource and enable tracking of specific objects, or ROIs. Image trackingby the image tracking unit, and in addition, a border adjustment by aborder adjusting unit 207, that are both configurable, is supported bythe region selecting unit 309 in one embodiment.

In certain embodiments of the invention, the first video source 307might be prerecorded static background scene or live video, providing avideo data from a remote location. It is also possible for the videosource 321 to be provided from (a stored video) from a video library.Selection of an image from one of the video source 307, 321 issupported, for example, with a keyboard, mouse or wireless remotecontrol.

Selection of region of interest within a selected image is conductedwithin the region selecting unit 109, either by manually orautomatically by highlighting a region of interest. In one relatedembodiment, the first video source 307 is prerecorded and the regions ofinterest are selected within region selecting unit 309 appropriately.

In another embodiment, the first video source 307 is a live feed from avideo camera, wherein the region selecting unit 309 selects certainaspects of the live feed, and the subtracting unit 311 deletes orsubtracts the selected regions of interest (there can be more than oneselected or specified). The output from the region subtracting unit 311is ultimately displayed on a display unit 313.

In another embodiment of the present invention, the video sources 307,321 comprise a video camera, the video data collected is processed suchthat a background is deleted and just a region of interest is retained.The video output is displayed on the display unit 313, saved forsharing, or broadcast to other remote video systems.

In another embodiment of the present invention, a video source 307provides video with multiple moving objects and background scene. Thevideo background subtractor system 305 also comprises of a storage unitand the mixing unit 319. The storage unit is required for storing videolayers created from the ROI extracted from the input video from thevideo source 307. The multiple moving objects are removed by astep-by-step processing technique thereby forming different layers ofvideo data, with each video layer storing one specific moving object. Itis then possible to remove, or replace, one moving object whileretaining the others if necessary—the region subtracting unit 311 isused for such removal of moving objects and the mixing unit 319 beingused for replacement, if needed. In addition, a moving object having ahigher motion or dominant motion may be removed and even replaced, whileretaining other moving objects in the video data from the video source307. It could also be confined or captured in its own individual videolayer.

In one embodiment, a moving object in the input video data from thefirst video source 307, with the highest motion parameters or dominantmotion parameters is removed and other moving objects having other typesof motion characteristics, perhaps slower, are retained and saved aspart of a video layer. If more than one object has the same unwantedmotion, then these objects can be removed simultaneously. The resultingvideo may be separated into a second video layer, and so on. Thus, theseresulting video for the individual moving objects, or sets of movingobjects, can be formed into forms different video layers, and saved asseparate video layers, and subsequently communicated as different videolayers. All the video layers may be subsequently combined in the mixingunit 319 to get the required output, wherein all the moving objectshaving different motion may be removed or otherwise manipulated. Theoutput of mixing unit is communicated to display unit 313 for display.

In one embodiment, the region selecting unit 309 and region subtractingunit 311 might be configured within a resolution adjusting capability,wherein the right resolution is employed for certain regions, asappropriate. For example, in the situations where a video source, suchas the first video source 307, may be a combination of video frommultiple spectral bands, such as video created by combining visible banddata and thermal IR band data, each band of video images may be capturedin a different layer, and different resolution technique may be appliedto the different layers. In addition, one of the layers may becompletely subtracted. Again, at least a portion of an image in onespectral band (layer) may be subtracted, and the resulting layers arecombined to created an output video stream that is displayed on thedisplay unit 319. The resolution of the different types of video data,in the different layers or spectral bands, can be adjusted as necessary.In some implementation, it might be desirable to adjust the resolutionof the subtracted region depending upon the user.

FIG. 4 is a schematic block diagram that illustrates an exemplaryembodiment of a region selecting unit 403 that acts as a control unit ofan exemplary video background subtractor system. The region selectingunit 403 comprises an image tracking unit 405, a border adjusting unit407 and a feedback control unit 409. It should be noted that theseelements provided in are just one embodiment of a control unit. Otherembodiments are contemplated wherein these elements are provided as partof a video manager 315, as part of the region selecting unit 309 or themixing unit 319.

The image-tracking unit 405 receives a video stream from a video source,such as the first video source 307. In one embodiment, the video streamis received after the region subtracting unit 311 has processed it. Theimage tracking unit 305 is configured to track a dynamic image such astalking person, a moving vehicle, a dancer in a video program of adance, etc. Replacement of a tracked image is supported by the regionsubtracting unit 311 after it has been tracked by the image trackingunit 405 and its border has been defined by the border adjusting unit407. The borders may be adjusted to make the selected ROI blend betterwith a new background, make the ROI more distinct from the background,or to extract the ROI out of the background.

In one embodiment addressing the broadcast of a video program of asporting event, such as a tennis match, the background is considered asstatic and the player is considered to be a moving region of interest.Thus, the tennis player moves on a background that is quite static, andin the image on the display unit, the tennis player is seen as movingaround on the screen. The image of the tennis layer is tracked and thetracked image or image data is provided by the image tracking unit 405to with border adjusting unit 407. The border adjusting unit 407 may beconfigured to adjust or “soften” the borders of the selected region ofinterest. The border adjustment is implemented in hardware, in softwareor in combination of the two.

The output of border adjusting unit 407 is provided, in certainembodiments, to a feedback control unit 409, that also receives thefeedback from the display unit 313 in addition to the video source, suchas via the second video source 121. This enables real-time adjustment inany image tracking or border adjusting. The feedback control is notnecessary in all embodiments.

FIG. 5 is a functional block diagram of a video processing system 503wherein the video subtracting system used to subtract a region ofinterest from a video data, such as a cable/satellite source 505, isincorporated into a set-top-box (STB) 507 or into a television, such asthe HDTV 509. The cable/satellite source 505 provides video data such asmovies, broadcast programs, etc. for reception by the STB 507 and/or theHDTV 509.

In one embodiment of the invention, the system 503 comprises a firstvideo source from cable/satellite link 405, which is communicativelycoupled to set-top-box 507. The signal from set-top-box 507 is deliveredto HDTV 509 and displayed on the full 4 corners of the screen of theHDTV 509, a remote control 511 and keyboard 513 being used to select,size and locate regions of interest in the center of the screen. TheHDTV 509, in a related embodiment, is capable of supporting thesubtraction of regions of interest using a video background subtractionsystem incorporated into the HDTV 509. In another related embodiment,the STB 507 is capable of supporting the subtraction of regions ofinterest using a video background subtraction system incorporated intothe STB 507.

A second video source, if required, can be obtained from the set-top-box507 that supports multiple video sources, including video data broadcastfrom the cable/satellite source 505. For example, in an application fortracking the dynamic activity of a tennis player in the game of tennisthat is broadcast, the video data stream consists of a relatively staticbackground and moving objects such as the tennis player. An imagetracking unit provided with the selecting unit of a video backgroundsubtractor system (incorporated into the STB 507, for example) is usedto track the dynamic activity of tennis players. After selecting dynamicactivity of the tennis player, the STB 507, or the HDTV 509 (as the casemight be) then applies a subtraction function, i.e., performssubtraction to generate a resulting video stream bounded by the dynamicactivity of tennis player. The subtracted video stream that comprisesdynamic motion and activity of the tennis player can be displayed on theHDTV 509 at different resolution, or with a different background,depending upon user requirement or preferences. Replace by an enhancedimage, which may be derived from the subtracted image/object.

Another example of a video processing system in image montagepreparation. Image montage or a mosaic image overcomes the limitationsof the limited field of view of a camera, by aligning and pasting framesin a video sequence. Since a camera's field of view is always smallerthen human field of view. In order to capture the entire scene, a numberof images can be captured and combined into a single image having largefield of view. In this case, for combining the two images, motionparameters between the images must be found and tracked. However, inmost practical situations the motion field is not homogeneous as theremay be several objects undergoing different motions. Moving objects canbias estimation of motion parameters because moving region of the imageindicates a transformation different than the transformation due to thecamera. Therefore these moving objects have to be removed or otherwiseneutralized. The image tracking unit of the selecting unit component ofthe video background subtractor system, that is part of the STB 507,and, in some embodiments, in the HDTV 509, is used to track the movingobjects. After a viewer selects moving objects, or based on predefinedselections of moving objects, the HDTV 509 then applies a subtractionfunction, performs subtraction to generate a resulting video streambounded by the background, where background is static. The subtractedimages (background images) are aligned and they are combined to form animage with a larger field of view, and displayed on HDTV at differentresolution depending upon user preferences.

FIG. 6 is a functional block diagram illustrating use of videoprocessing system for overlaying two pure unrelated video streams,wherein a set-top-box (STB) 617 or a television (TV) display comprises avideo background subtractor system and wherein a user can specify aregion of interest using a remote control or keyboard. The videoprocessing system 605 comprises a cable/satellite broadcastingconnection (interface) 607 that behaves as a first video source, asecond video source 627, the STB 617, the TV 609, a remote control 611and a keyboard 613. It also comprises an interface to the Internet 619and a remote media storage 621.

A live feed video streams from the cable/satellite box 605 is receivedby the STB 617. The video data transmitted from the satellite or a cableTV connection is provided to the STB 617. The STB delivers it to thetelevision 609, which is an HDTV 609 in one embodiment. The receivedsatellite or a cable TV programs are typically displayed on the full 4corners of the screen of the TV 609. The remote control 611 and keyboard613 are associated with the HDTV 609, or with the set-top-box 617, andthey are used to program channels, as well as to select, a size andlocate a region of interest on the screen. The region of interest couldbe shaped in a number of geometrical shapes—circular, oval, rectangular,heart shaped or any other region based on user preferences.

In one embodiment, after a user selection of a region of interest, TV609 applies a subtracting function, performs subtraction to generate aresulting video stream bounded by the region of interest. This resultingvideo stream is overlayed on to another video stream, if necessary. Forexample, a first video data received through the cable/satellitebroadcasting connection 607 can be a data presented in a region that isshaped like a heart, for example, while a video program from the secondvideo source 625, 621 is being displayed on the TV 609. Thus, a regionof interest containing dynamic video data from cable/satellitebroadcasting connection 607 is combined with video data from the secondvideo source 625, 621, and the region of interest displayed is shapedlike a heart. The position of the region of interest can bemanipulated—it could be at the center of the TC 609 screen, or moved toa side, for example.

The second video source 625 could be a stored data in the video libraryor a remote media storage 621 that is linked to the TV 609 screen viathe Internet 619. The second video source 625, for example, may be astatic data that needs to be enhanced with a dynamic data. To enhancethe static video data of the second video source with a dynamic contentfrom the cable/satellite broadcasting connection 607, the remote control611 is used to select, size and locate the heart shape to a particularposition on the TV 609 screen.

An exemplary process of selection of a ROI involves selection ageometrical shape, such as a heart shape or a rectangle while the firstvideo data from the cable/satellite broadcasting connection 607 is beingviewed, such selection being conducted with a remote control or akeyboard. When a viewer selects a large heart shaped region of interestfrom the first source, the TV 609 then applies a subtraction function,performs subtraction to generate a resulting video stream bounded by theheart shape. This heart shaped subtracted region from the first videosource is then overlayed on the second video source selected by theviewer, and the combined output is displayed on the TV 609 screen.

FIG. 7 is a schematic block diagram showing an exemplary videobackground subtractor system 705 that is used for transmitting acomposite video signal incorporating a plurality of video sources 707,721, 725. The video background subtractor system 705 comprises a regionselecting unit 709 to which multiple video sources are communicativelycoupled, such as a first video source 707, a second video source 721 anda third video source 725. The video background subtractor system 705also comprises a mixing/superimposing unit 719 that is capable ofcombining ROIs from one or more sources into a composite output. Themixing/superimposing unit 719 is also capable of combining ROIs withvideo data from one of the video sources, such as the first video source707. The output from the mixing/superimposing unit 719 is eitherdisplayed on a display unit 713, stored locally or remotely, ortransmitted via a transmitting interface 727.

In one embodiment, for example, for certain types of broadcast videoimplementations, the output of the video background subtractor system705 is a transmitted video signal, which is ultimately displayed on aremote display unit, such as 713. A live telecast of a conversationbetween two users could comprise a first video source 707 containing theimage or video data of a first speaker, a second video source 721containing an image or a video data of a second speaker, and a thirdvideo source 725 could be a video data associated with a stage or studiobackground. The selected regions of interest from first video source 707is a ROI around the image of the first speaker, the selected region ofinterest from the second video source 121 is an ROI around the image ofthe second speaker. The selecting unit 709 is used to select the ROIswith the images of the first and second speakers, and the background isselected from the third video source 725. A region subtracting unit 711is used to generate the necessary ROI with the first and the secondspeakers from the associated video sources after the selection of theROI. The mixing unit/superimposing unit 719 is used to create acomposite output. Before the composite output can be transmitted ordisplayed, there is a need to apply shading control, and contrast/borderadjustment to the images. This is provided by the region subtractingunit 719 in a related embodiment. The mixing unit/superimposing unit 719place the images in the appropriate locations in the appropriate videosource that forms the background, and composite output, that would thenbe received by users or viewers at remote destinations, is transmittedvia the transmitter interface 727. The net result on the compositeoutput is the seamless juxtaposition of the images of the two speakerson a required third background with the expected impression on theviewers being that the speakers are next to each other in closeproximity. In fact, it would appear to the viewers that the two speakerswere in the same room or the same studio, having a face-to-faceconversation, even though they could, in reality, be actually in remotelocations miles apart. A fourth or fifth video source could be provided,as necessary, which could provide images of a moderator, or other scenesor other individual participants of an event.

FIG. 8 is a schematic block diagram that illustrates an exemplaryembodiment of a set-top-box 807 that comprises the typical components ofthe video background subtraction system, such as a region selecting unit809, a region subtraction unit 811, a mixing unit 813. The STB 807 alsocomprises a video manager 815 and local video storage 817 and anadditional video source 827, in addition to typical STB components suchas a processing circuitry 819 and a communication interface 825. Theprocessing circuitry 819 comprises a multi-driver control 821 and atranscoding component 823.

For operation in a real time environment, a user using a remote controlor the set-top-box 807 based on a configuration information can selectthe region of interest from the video source while the video data isbeing provided to the region selecting unit 809. Utilizing conventionalinput and control devices such as keyboard, remote control, a user cancontrol the region selecting unit 809. The regions of interest, whenselected, are specified or captured in terms of coordinates in the areasof a screen, vectors, etc. The region of interest can also be selectedemploying predefined objects representing ROIs. Such predefined ROIspecification can be used for processing both static or dynamic videodata. In addition, the region selecting unit 809 can be configured totrack or to follow a dynamic region of interest, such as a talkingperson, a moving person, etc. The mixing unit 813 can be configured tomix the region of interest from the video source such as the video datafrom a cable or satellite connection, with the video data obtained fromthe local video storage 817 or from the additional video source 827. Themixing unit 813 can also be configured to superimpose information fromthe cable or satellite connection on to video data from the local videostorage 817 or from the additional video source 827. One example of suchsuperimposition might be the utilization of ROIs derived from the videodata provided by cable/satellite connection in creating a combinedoutput wherein these ROIs are superimposed upon a background sceneobtained from the additional video source 827. The background scene canbe a mountain scene or a studio scene, obtained from the additionalvideo source 827. In this case the required region of interest from thevideo data provided by cable/satellite connection is separated from it'sbackground using background subtraction techniques. Then the requiredregion of interest are superimposed using the mixing unit 813 onto a newbackground scene obtained from the additional video source 827. Thesuperimposed video output is displayed on a TV or stored in the localvideo storage 817.

While the present invention has been described with reference to certainembodiments, it will be understood by those skilled in the art thatvarious changes can be made and equivalent may be substituted withoutdeparting from the scope of the present invention. In addition manymodifications may be made to adopt a particular situation or material tothe teachings of present invention without departing from its scope.Therefore, it is intended that the present invention not be limited toparticular embodiment disclosed, but that the present invention whichinclude all embodiments following within the scope of the appendedclaims.

1. A video background subtractor system comprising: a first video sourcethat provides a first video data that comprises a first background; aregion selecting unit communicatively coupled to the first video sourcefor selecting a first region of interest from the first video data anddistinguishing it from the first background; a region subtracting unitcommunicatively coupled to the region selecting unit for subtracting thefirst region of interest from the first video data to create a firstoutput comprising the first background; and the region subtracting unitsubtracting the first background from the first video data to create asecond output comprising the first region of interest; and wherein thefirst video data comprises a plurality of layers of video data, whereineach of said plurality of layers corresponds to different ones of acorresponding plurality of moving objects, and wherein the regionselecting unit selects the first region of interest in one of theplurality of layers; and the region subtracting unit subtracting thefirst region of interest from the first video data.
 2. The videobackground subtractor system of claim 1, further comprising: a secondvideo source that provides a second video data; and a mixing unit thatreplaces the first region of interest with at least a portion of thesecond video data to create a combined output.
 3. The video backgroundsubtractor system of claim 2, further comprising: a mixing unit capableof mixing at least a portion of the first video data with at least aportion of the second video data to create a combined output; the mixingunit superimposing the second output onto the second video data tocreate a combined output.
 4. The video background subtractor system ofclaim 1, further comprising: a second video source that provides asecond video data, the second video data comprising a second background;the region selecting unit communicatively coupled to the second videosource selecting a second region of interest from the second video dataand distinguishing it from a second background; the region subtractingunit subtracting the second region of interest from the second videodata to create a second output comprising the second background.
 5. Thevideo background subtractor system of claim 4, further comprising: thevideo background subtractor system providing a composite outputcomprising the first region of interest, the second region of interestand at least one of the first background and the second background. 6.The video background subtractor system of claim 5, wherein the videobackground subtractor system provides a composite output comprising thefirst region of interest and the second region of interest.
 7. The videobackground subtractor system of claim 1, further comprising: an inputdevice that is one of a remote control, a mouse and a keyboard for theselection of the user selected region of interest by a viewer; theregion subtracting unit selecting the user selected region of interestfrom the first video data and distinguishing it from the firstbackground.
 8. The video background subtractor system of claim 1,wherein, the region selecting unit selects a plurality of regions ofinterest from the plurality of layers of video data; and the regionsubtracting unit subtracts at least one of a plurality of regions ofinterest in the at least one of the plurality of layers of video data tocreate the combined output.
 9. The video background subtractor systemaccording to claim 8, further comprising a mixing unit coupled to theregion subtracting unit, the region subtracting unit subtracting atleast one of a plurality of regions of interest and the mixing unitmixing the others of the plurality of layers of video data to create acombined output.
 10. A video processing system for subtracting a regionof interest from an input video data obtained from a first video source,the video processing system comprising: the first video source thatprovides the input video data; a region selecting unit communicativelycoupled to the first video source for selecting a first region ofinterest from the input video data and distinguishing it from a firstbackground; a region subtracting unit communicatively coupled to theregion selecting unit for subtracting the first region of interest fromthe input video data to create a first output comprising the firstbackground; and the region subtracting unit subtracting the firstbackground from the first video data to create a second outputcomprising the first region of interest; and wherein the first videodata comprises a plurality of layers of video data, wherein each of saidplurality of layers corresponds to different ones of a correspondingplurality of moving objects, and wherein the region selecting unitselects the first region of interest in one of the plurality of layers;and the region subtracting unit subtracting the first region of interestfrom the first video data.
 11. The video processing system of claim 10,wherein the first region of interest is dynamic, the video processingsystem further comprising: an image tracking unit for tracking the firstregion of interest in the input video data from the first video sourcewherein the image tracking unit is communicatively coupled to the regionsubtracting unit.
 12. The video processing system of claim 11, furthercomprising: a border-adjusting unit connected to the region subtractingunit for adjusting the borders of a region of interest when it isselected for display on a display unit.
 13. The video processing systemof claim 12, further comprising: a feedback control unit for adjustingthe at least one of the first output and the second output that isdisplayed on the display unit.
 14. The video processing system of claim10, wherein the first video source comprises a live feed from an imagecapture device.
 15. The video processing system recited in claim 10,wherein the first video source comprises a live feed from a remotelocation.
 16. The video processing system recited in claim 10, whereinthe input video data comprises a relatively static background withmultiple moving objects, each of the multiple moving objects beingselected as a region of interest, at least one of which is replaced orsubtracted by the region subtracting unit to generate the first output.17. The video processing system recited in claim 10, wherein the regionselecting unit employs an input device for the specification of theregion of interest by a viewer, the input device being one of akeyboard, a mouse, a remote pointing device, a tablet and a touchscreen.
 18. The video processing system recited in claim 10, wherein theinput video data comprises one or more moving objects, each of theobjects being selected as a region of interest, at least one of which isreplaced or subtracted by the region subtracting unit to generate thefirst output.
 19. The video processing system recited in claim 17,wherein one or more objects may be combined to form the region ofinterest.
 20. The video processing system recited in claim 10, whereinit is capable of transmitting the region of interests in differentlayers.
 21. A method of operating video processing system, the methodcomprising: receiving a first video data from a first video source;selecting, a region of interest from the first video data; subtractingthe selected region of interest from the first video data to create amodified first video data; and displaying the modified first video dataas output; and wherein the first video data comprises a plurality oflayers of video data, wherein each of said plurality of layerscorresponds to different ones of a corresponding plurality of movingobjects, and wherein the region selecting unit selects the first regionof interest in one of the plurality of layers.
 22. The method recited inclaim 21, wherein the first video source is one of an image capturedevice, a cable TV broadcast connection, a satellite TV broadcast, avideo game input and a prerecorded video clip from a local storage.